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REMARKS 

Applicant would like to thank the Examiner for the 
thorough examination of the present application. Formal 
drawings were filed October 10, 2003. A substitute 
specification is enclosed herewith in which the line spacing 
are now double spaced. A noted grammatical error has also 
been corrected in the specification. 

The Examiner has taken the position that the 
specification does not provide support for the first and 
second filtering algorithms as recited in Claim 50, and as a 
result, the Examiner has also taken the position that Claim 50 
is indefinite. Support in the specification may be found on 
page 15, lines 3-18. Consequently, the Applicant submits that 
Claim 50 is definite. 

As noted by the Examiner, Claims 4 2 and 4 3 are 
dependent from Claim 36. As helpfully noted by the Examiner, 
Claim 36 has been amended to more clearly define the steps of 
storing and filtering. Claim 42 recites that the output of 
the first filtering algorithm provides the defective pixel 
values, and Claim 43 has been amended to more clearly define 
how a defective pixel value is determined. The Applicant 
submits that Claims 42 and 43 are definite in view of amended 
Claim 36. 

Claims 51-64 have been cancelled, and new Claims 65- 
84 have been added. New independent Claim 65 is based upon 
Claims 33 and 36. New independent Claim 82 is based upon 
cancelled Claims 55, 56, 58 and 59. The arguments supporting 
patentability of the claims are presented in detail below. 
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I. Independent Claim 33 Is Patentable 

The Examiner rejected independent Claim 3 3 over the 
Maruo patent. The present invention, as recited in 
independent Claim 11, is directed to a method for processing a 
video data stream in an electronic imaging system. The video 
data stream comprises a series of pixel values corresponding 
to pixel sites in the electronic imaging system. The method 
comprises the step of filtering the video data stream in real 
time for correcting/modifying defective pixel values. The 
present invention advantageously allows defective pixel values 
to be corrected/modified after manufacture of an electronic 
imaging device that processes a video data stream. 

Referring now to Maruo, a method of image processing 
is disclosed. In particular, column 1, lines 5-11 of Mauro 
discloses that the image processing may be used in an image 
quality inspection unit such as an LCD tester or a CCD tester 
in which an image is used to inspect an object being examined. 
Mauro further discloses that the image processing may be used 
in an image recognition apparatus in which an object being 
examined is recognized through an image thereof . 

The Examiner has taken the position that Mauro is 
not limited to still images, and that the image processing may 
also be applied to a digital video stream as well. The 
Applicant respectfully disagrees because Mauro fails to teach 
or suggest that the image processing may be applied to a 
digital video stream. 

Instead, Mauro discloses that input digital image 
data is subject to a Wavelet transform on a screen on which 
the image is formed, and an image quantity for a combined area 
of X-axis high pass information and Y-axis high pass 
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information contained in the image data which results from the 
Wavelet transform is calculated. An object being examined is 
determined to be acceptable or faulty depending on whether the 
image energy quantity is below or above a given value. 

As illustrated in the flow diagrams of FIGS. 1 and 
6, the digital image is first stored in memory before being 
process. Mauro does not even mention "in real time" anywhere 
through the patent. In fact, Mauro teaches away form 
correcting/modifying defective pixel values in a video data 
stream or even in a digital still image. Reference is 
directed to column 3, lines 43-50 of Mauro, which provides: 



"In accordance with the invention, a 
digital image data which is input is 
subject to a Wavelet transform on a screen 
on which the image is formed, an image 
energy quantity is determined for a 
combined area of X-axis and Y-axis high 
pass information contained in the Wavelet 
transformed image data, and an inspection, 
recognition and/or decision is performed 
or rendered for an object being examined 
on the basis of the image energy quality." 
(Emphasis added.) 

In other words, the image processing in accordance 
with the Mauro patent is directed to examining objects for 
inspection or recognition purposes. The Examiner has taken 
the position that column 2, lines 41-65 of Mauro discloses 
that a pixel value is corrected/modified. 

However, reference is directed to lines 52-55 which 
provides that "it will be seen that active pixels in the 
binary image D which have the pixel value 1 may represent a 
noise, a point defect or a component of a lineal or area 
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defect." Reference is also directed to lines 61-63 which 
provides "Accordingly, an image processing operation is 
applied so as to remove an isolated point, namely, a pixel in 
question which has the pixel value of 1." 

The isolated point is not corrected. Instead, it is 
used along with other isolated points to determine if the 
object being examined has an area defect . Determining if an 
object has a defect area is completely different than 
filtering a video data stream in real time for 
correcting/modifying defective pixel values, as recited in 
independent Claim 33. Accordingly, it is submitted that 
independent Claim 33 is patentable over Mauro. 

II. New Independent Claim 65 Is Patentable 

New independent Claim 65 is based upon Claims 33 and 
36, which the Examiner rejected as being unpatentable over 
Maruo in view of Ninomiya. Independent Claim 65 is directed 
to method for processing a video data stream in an electronic 
imaging system comprising a memory, and the video data stream 
comprises a series of pixel values corresponding to pixel 
sites in the electronic imaging system. 

The method comprises filtering the video data steam 
in real time for correcting/modifying defective pixel values. 
In particular, the filtering comprises filtering pixel values 
not stored in the memory using a first filtering algorithm, 
identifying defective pixel values, storing locations of the 
defective pixel values in the memory, and filtering the 
defective pixel values stored in the memory using a second 
filtering algorithm. As noted above, the present invention 
advantageously allows defective pixel values to be 
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corrected/modified after manufacture of an electronic imaging 
device that processes a video data stream. 

Referring now to Maruo, the deficiencies as noted 
above also apply here. Essentially, Maruo fails to teach or 
suggest that the disclosed image processing may be applied to 
a digital video stream as in the claimed invention. In 
addition, Maruo fails to teach or suggest that defective pixel 
values are corrected/modified as in the claimed invention. 
Instead, the image processing in Mauro is directed to 
examining objects for inspection or recognition purposes. 
Mauro thus fails to even disclose a need for correcting/ 
modifying defective pixel values. 

The Examiner cited the Ninomiya patent as disclosing 
a memory for storing defective pixel values. However, 
Ninomiya fails to provided the noted deficiencies with respect 
to Maruo. Accordingly, it is submitted that independent Claim 
65 is patentable over Mauro in view of Ninomiya. 

III. New Independent Claim 82 Is Patentable 

New independent Claim 82 is based upon Claims 55, 
56, 58 and 59, which the Examiner rejected as being 
unpatentable over Lougheed in view of Maruo and in further 
view of Kaplan. Independent Claim 82 is directed to an 
apparatus for processing a video data stream comprising an 
electronic imaging device, and a first filter circuit 
connected to the electronic imaging device for filtering the 
video data stream in real time for correcting/modifying 
defective pixel values. The video data stream comprises a 
series of pixel values corresponding to pixel sites in the 
electronic imaging device. 
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A sampling circuit is connected to the first filter 
circuit for sampling the video data stream to obtain a data 
set comprising a current pixel value and a plurality of 
adjacent pixel values. A ranking circuit is connected to the 
sampling circuit for sorting the plurality of adjacent pixel 
values into a rank order based upon predetermined criteria. A 
comparator is connected to the ranking circuit for comparing a 
current pixel value with the plurality of adjacent pixel 
values of selected ranks, and for generating a first filter 
output based upon the comparison. A median circuit is 
connected to the ranking circuit for determining a median 
value of the plurality of adjacent pixel values and for 
generating a second filter output equal to the median value. 

Referring now to Lougheed, an apparatus for 
filtering digital image matrices is provided. In particular, 
a neighborhood transformation stage performs filtering 
operations on an image matrix. A group of pixel values in the 
image matrix are accessed. A series of comparators are used 
to compare the values of neighboring pixels with the value of 
a given pixel in the group. The outputs of the comparators 
are combined to form a vector to address a programmable look- 
up table. The addressed output of the look-up table is used 
as a control signal for a selector which either maintains the 
value of the given pixel or transforms it into another value 
associated with a neighboring pixel whose value has been 
ranked in a predetermined order by a rank sorter device used 
to compare the values of the neighboring pixels with each 
other . 

The Applicant respectfully submits that Lougheed is 
directed to still images and does not apply to a video data 
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stream. As recited in independent Claim 82, the first filter 
circuit is connected to the electronic imaging device for 
filtering the video data stream in real time for 
correcting/modifying defective pixel values. Lougheed fails 
to teach or suggest that the image processing may be applied 
to a digital video stream in real time. 

As correctly noted by the Examiner, Lougheed fails 
to disclose a median circuit connected to the ranking circuit 
for determining a median value of the plurality of adjacent 
pixel values and for generating a second filter output equal 
to the median value. The Examiner cited Maruo as disclosing 
additional filtering of defective pixels after an initial 
filtering step. The deficiencies as noted above with respect 
to Maruo also apply here. 

Maruo fails to teach or suggest that the disclosed 
image processing may be applied to a digital video stream as 
in the claimed invention. In addition, Maruo fails to teach 
or suggest that defective pixel values are corrected/modified 
as in the claimed invention. Instead, the image processing in 
Mauro is directed to examining objects for inspection or 
recognition purposes. Mauro also fails to disclose a need for 
correcting/modifying defective pixel values since it is 
primarily concerned with identifying defect areas in objects. 

The Examiner cited Kaplan as disclosing afiltering 
algorithm that replaces the current pixel value with a median 
value of the plurality of adjacent pixel values. However, 
Kaplan also fails to provide the deficiencies as noted above 
with respect to Lougheed and Maruo. Consequently, even if the 
references were combined as suggested by the Examiner, the 
claimed invention is still not produced. Accordingly, it is 
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submitted that independent Claim 82 is patentable over 
Lougheed in view of Maruo and in further view of Kaplan. 

In view of the patentability of independent Claims 
33, 65 and 82, it is submitted that the dependent claims which 
recite yet further distinguishing features of the invention 
are also patentable. These dependent claims need no further 
discussion herein . 
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CONCLUSION 

In view of the arguments provided herein, it is 
submitted that all the claims are patentable. Accordingly, a 
Notice of Allowance is requested in due course. Should any- 
minor informalities need to be addressed, the Examiner is 
encouraged to contact the undersigned attorney at the 
telephone number listed below. 



Respectfully submitted, 



MICHAEL W. TAYLOR 7^ 
Reg. No. 43,182 

Allen, Dyer, Doppelt, Milbrath 

& Gilchrist, P. A. 
255 S. Orange Avenue, Suite 1401 
Post Office Box 3791 
Orlando, Florida 32802 
407-841-2330 
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I hereby certify that this correspondence is being 
deposited with the United States Postal Service as first class 
mail in an envelope addressed to: MAIL STOP NON-FEE AMENDMENT, 
COMMISSIONER FOR PATENTS, P.O. BOX 1450, ALEXANDRIA, VA 22313- 
1450, on this day of October, 2003. 
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DEFECT CORRECTION IN ELECTRONIC IMAGING SYSTEMS 



Field of the Invention 

The present invention relates to electronic 
imaging systems, and, more particularly, to a method 
and apparatus for correcting defects in video data 
5 generated by an electronic imaging system. 



Background of the Invention 

A majority of electronic imaging devices are 
now implemented using semiconductor technologies. 
Examples include the charge coupled display (CCD) , 

10 which is implemented using a MOS manufacturing process, 
and, more recently, image sensors manufactured using 
standard CMOS semiconductor processes. In all of these 
cases, the sensor normally includes a one or two 
dimensional array of discrete pixels. As a result of 

15 the manufacturing processes employed in the production 
of such devices, occasional defects occur at individual 
pixel sites. Such defects may cause the affected pixel 
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to be brighter or darker than the true image at that 
point, including the extreme cases of saturated white 
or black pixels. 

These defects affect some proportion of the 
5 plurality of individual imaging devices or chips on 
each manufactured wafer. The chips affected must 
normally be rejected unless the defects can be masked 
or corrected. It is more economical to mask or correct 
defective pixels, thus enabling otherwise rejected 

10 chips to be passed. This improves the apparent yield 
of good imaging chips per wafer, and, thereby lowers 
the cost per usable chip. It is known in the art to 
calibrate imaging devices at the point of camera 
manufacture so that the locations of defective pixels 

15 in the imaging array are identified and stored. In 
subsequent use of the device, pixel data from these 
locations are masked or corrected in the live video 
data stream. 

One simple and well known masking technique 

20 is to substitute the defective data with a copy of the 
value of a neighboring or adjacent pixel. More 
sophisticated techniques are also possible, and 
typically may produce an estimate of the correct value 
of the defective pixel data. This is done by applying 

2 5 an algorithm to the data obtained from the neighboring 
pixels in one or two dimensions. Generally, the best 
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correction filters use a mixture of linear and non- 



linear estimators and work on at least a 3 x 3 pixel 



neighborhood centered on the defective pixel . 



This prior technique of calibrating 



5 individual sensors at the point of manufacture has two 
main disadvantages. First, and most significantly, the 
process of calibrating the sensor to determine defect 
locations is an inconvenient and expensive 
manufacturing burden. Second, defects may sometimes be 

10 transient in nature, so that defects present and 
corrected for at the time of calibration may 
subsequently disappear, or worse, new defects may occur 
subsequent to calibration. These latter defects will 
remain uncorrected in subsequent camera use and will 

15 result in blemishes on the images output by the camera. 



with the correction of defects arising from defective 
pixel sites in electronic image sensors, and is also 
20 applicable to a more general noise reduction in video 
data streams. The invention is equally applicable to 
monochrome and color video data and may be useful in 
still imaging systems as well as kinematic video 
systems . 



Summary of the Invention 



The invention is most particularly concerned 
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A first object of the present invention is to 
provide a method and an apparatus for the correction of 
defects in an electronic imaging system which prevents 
or reduces the above mentioned disadvantages of prior 
5 art image defect correction schemes. 

While the invention may be implemented using 
known error correction algorithms for correcting the 
pixel values output by defective pixel sites, it is a 
further object of the present invention to provide an 
10 improved method and apparatus for filtering video data 
signals, both for the purpose of correcting image 
defects originating from defective pixel sites and for 
more general noise reduction purposes. 



Brief Description of the Drawings 

15 Embodiments of the invention will now be 

described, by way of example only, with reference to 

the accompanying drawings, in which: 

Fig. 1 is a block diagram illustrating a 

first embodiment according to the present invention; 
20 Fig. 2 is a block diagram illustrating a 

preferred embodiment according to the present 

invention; 

Figs. 3(a) and 3(b) are illustrations 
representing pixel neighborhood locations used in 
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correcting image defects according to the present 
invention; 

Fig. 4 is a more detailed block diagram 
according to the present invention using the pixel 
5 neighborhood location representations illustrated in 
Fig. 3; and 

Fig. 5 is a graph illustrating operation of a 
digital filter used in the embodiment illustrated in 
Fig. 4. 



10 Detailed Description of the Preferred Embodiments 

Referring now to the drawings, Fig. 1 
illustrates a first, most general embodiment of the 
invention. An image sensor 10 of a known type 
comprises an array of pixels. The sensor array 10 

15 outputs an analog data stream which is converted to 
digital form by analog to digital conversion means 12 
or converter. Assuming a two dimensional pixel array, 
the data stream comprises a series of pixel values 
output line by line from the sensor 10. The digital 

2 0 data stream would normally be encoded by encoding means 
14 or an encoder in a manner to suit the intended end 
use of the video data. 

In accordance with the present invention, the 
live video data stream is filtered in real time by 

25 digital filter means 16 or a filter to correct or mask 
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anomalous pixel values which are judged to arise from 
defective pixel sites in the sensor 10. Typically, the 
filter 16 judges a pixel value to be defective if it is 
significantly higher or lower than its neighbors or 
5 adjacent pixels in either one or two dimensions. The 
filter replaces the defective pixel value with a 
substitute value. The substitute value may be derived 
by any suitable algorithm, which may involve linear 
and/or non- linear processes which may operate on 

10 surrounding pixel data from a one or two dimensional 
neighborhood surrounding the defective pixel value. 

The filter 16 works permanently on the normal 
sensor output and does not require the use of any 
reference scene or predetermined calibration data. 

15 Rather, the filter depends on predetermined criteria 

for identifying defective pixel values in the live data 
stream and on predetermined rules for deriving 
substitute pixel values to replace the defective pixel 
values . 

20 This live or in-line correction of defective 

pixels overcomes the manufacturing burden of prior art 
techniques and deals automatically with defects which 
arise after manufacture. It further provides a degree 
of noise filtering on noisy images, correcting 

25 excessively large single-pixel noise spikes. Applying 
automatic correction in this way to an entire image 
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can, in some circumstances, cause an undesirable 
deterioration in the overall image quality unless the 
parameters parameters of the correction filter are 
relaxed. This 1 imits the effectiveness of the 
5 technique in its most basic form. 

A suitable class of a pixel -correcting filter 
is one which uses the central pixel data itself as part 
of the data set used to determine the correction to be 
applied. Typically, this means that the non-defective 

10 portions of the image, i.e., the majority of each 

image, are unaffected by the presence of the correcting 
filter. The filter will, however, correct defects of 
large magnitude. 

Unfortunately, many defects which would be 

15 desirable to correct are not of large magnitude. 

Typical examples are pixels with a significant gain 
error, or pixels which are stuck at an intermediate 
image value. A single filter capable of correcting 
these more subtle defects while not falsely correcting 

20 non-defective pixels causing an undesirable effect on 
the overall image, such as by producing a smearing 
effect, has not been developed. 

Fig. 2 illustrates a preferred embodiment of 
the invention, in which the single filter 16 of Fig. 1 

25 is replaced by first and second filter stages 18 and 22 
and a defect memory or database 20. In accordance with 
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this scheme, the first filter stage 18 performs two 
functions. First, it applies a more subtle correction 
algorithm to the complete data stream to correct 
defects of lower magnitude as noted above. Second, it 
5 identifies pixels exhibiting more extreme defects, and 
passes information regarding these pixels to the defect 
memory 20. The defect memory 20 stores information 
regarding those pixels which are judged to be most 
severely defective. The defect memory 20 controls the 

10 operation of the second filter stage 22, which applies 
more severe correction selectively to those pixels 
identified in the defect memory 20. 

Typically, the number of pixels for which 
severe correction is required will be less than 1% of 

15 the total pixel count. The pixel locations stored in 
the defect memory 20 are restricted to those that, 
historically, appear to be most severely in error as 
detected by the first filter stage 18. That is, for 
each video frame or for each still image captured by 

20 the sensor, all defects are monitored by the first 

filter stage 18. Those pixel locations exhibiting the 
largest apparent errors are added to the defect memory 
20 if not already identified and stored. 

To enable the contents of the defect memory 

2 5 2 0 to remain dynamic over time, a management strategy 
is required so that locations representing transient 
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noise defects or defects which disappear over time can 
be identified and removed from the defect memory 20. 
Besides preventing future correction of non-defective 
pixel values, this also creates memory space for new or 
5 previously undetected defects. The memory space 20 is 
necessarily limited, and it is desirable that it be as 
small as possible consistent with the number of defects 
which are likely to be encountered in practice. 
Typically, the defect memory 20 might store less than 

10 1% of all possible pixel locations. Accordingly, no 
more than 1% of pixels will be subject to severe 
correction. This proportion is so low as to be 
unnoticeable to a human observer of the corrected video 
or still image. 

15 A preferred embodiment of the scheme 

illustrated in Fig. 2 will now be described with 
reference to Figs. 4 and 5. Referring first to Figs. 
3 (a) and 3 (b) , these illustrate examples of pixel 
neighborhoods operated on by digital filters of the 

2 0 type employed in the invention. In a two dimensional 
pixel array, each pixel is surrounded by eight 
immediately neighboring pixels forming a 3 x 3 array. 
The pixels at the edges of the array are neglected. 

The particular pixel operated on by a filter 

25 at any point in time is the central pixel p(c) of the 3 
x 3 array. Fig. 3(a) illustrates the situation when 
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the filter includes the central pixel value along with 
the values of the surrounding eight pixels in the data 
set employed to determine a substitute value for p(c). 
Fig. 3(b) illustrates the situation when the filter 
5 excludes the central pixel value from the data set 
employed to determine a substitute value for p(c) . 
These two alternatives are both employed in the two 
stage filtering provided by the preferred embodiments 
of the present invention, as described in greater 

10 detail below. It will be understood that the use of a 
3X3 array for the filter data set is merely an 
example being particularly applicable to monochrome 
image sensors. Larger and/or differently oriented 
arrays may be appropriate in some circumstances, 

15 particularly for color sensors. The approach described 
in the present example can clearly be extended to other 
shapes or other array sizes. 

Referring now to Fig. 4, there is shown a 
block diagram of a video data filtering system 

20 corresponding to blocks 18, 20 and 22 of Fig. 2. The 
input data stream includes a series of input pixel 
values p(in), and the output data stream includes a 
series of output pixel values p(out) . 

The input data stream is first sampled by a 

25 sampling network comprising line memory buffers 30 and 
32, each of which is capable of storing a complete line 
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of video data. The input data stream is also sampled 
by individual pixel value memory buffers 34, 36, 38, 
40, 42 and 44. The incoming video signal is routed 
through the line buffers 30, 32 and into the pixel 
5 buffers 34-44 so that, over a number of clock cycles, 
nine pixel values for the central pixel p(c) and 
surrounding neighbors are accumulated to be operated on 
by the filter system. The line buffers 30, 32 suitably 
comprise random access memory, while the pixel buffers 

10 34-44 may be D-type flip-flops. 

The central pixel value p(c) is extracted on 
line 46 as shown, while the eight neighboring values 
are applied to block 48. Block 48 sorts the values of 
the neighboring pixels into rank order according to 

15 their amplitudes. Block 48 also outputs the values in 
rank order, with the highest value output on the upper 
output line 48U and the lowest value on the lower 
output line 48L. In this example, the filter system 
only employs the highest, lowest and middle two ranking 

2 0 values out of the eight input values. However, 
variations on this example could utilize other 
combinations of the eight ranked values, as shall be 
discussed below. 

The ranked values of the neighboring pixels 

25 are employed by both the first and second stage filter 
processes 18 and 22 of Fig. 2. The two filter stages 
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share components and functions of the embodiment 
illustrated in Fig. 4, rather than being discrete 
systems as shown in Fig. 2. However, their essential 
functionality is separate and is in accordance with the 
5 schematic representation provided by Fig. 2. The first 
stage filtering operates to apply relatively subtle 
correction to the entire data stream while at the same 
time identifying defect locations to which the second 
stage filtering is to be applied, as follows. 

10 The highest and lowest ranked pixel values on 

lines 48U and 48L and the central pixel value p(c) on 
line 46 are input to block 50, which operates as a 
three to one multiplexer. Block 50 compares p(c) with 
the highest and lowest ranked values. If the value of 

15 p(c) is greater than the highest ranked value, then the 
highest ranked value is output from block 50, replacing 
p(c) in the data stream. If the value of p(c) is less 
than the lowest ranked value, then the lowest ranked 
value is output from block 50, replacing p(c) in the 

20 data stream. If the value of p(c) is less than the 

highest ranked value and greater than the lowest ranked 
value, or is equal to either value, then the value of 
p(c) is output from block 50 so that p(c) is unaffected 
by the first stage filter . 

25 This filtering scheme is illustrated in Fig. 

5, in which the rank of the input pixel value is 
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plotted against the rank of the pixel value which is 
output by the filter. The nine ranks of this example 
are numbered from -4 to +4, with zero being the rank of 
the median pixel value. The graph shown corresponds to 
5 the scheme described above. If p(c) is ranked +4 then 
it is replaced by the value of rank +3. If p(c) is 
ranked -4 it is replaced by the value of rank -3. 
Otherwise, it is unaffected by the filter. 

The filter could be modified to allow maximum 

10 values restricted to ranks 1 or 2, as indicated by the 
dot-and-dash lines, in which case different outputs 
from block 48 would be employed. The filter could also 
be made to be switchable between these different modes 
of operation if required. The horizontal axis of Fig. 

15 5 corresponds to a median filter, in which the median 
value is output regardless of the input value. The 
diagonal line through the origin indicated by the 
dashed line corresponds to zero filtering, in which the 
output is always equal to the input. 

20 Since this filtering operation is applied to 

the entire data stream, it acts as a general noise 
reduction filter as well as correcting relatively 
subtle defects arising from defective pixel sites in 
the sensor array. It is potentially useful in 

25 applications other than that illustrated in Figs. 2 and 
4. For example, it could be employed purely as a noise 
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reduction filter in imaging systems using prior art 
calibration schemes to correct sensor defects. This 
filtering scheme will be referred to hereinafter as a 
scythe filter and its output value as the scythe value, 
5 or may simply be referred to as the filter and filter 
value . 

The second stage filtering 22 of Fig, 2, in 
this example, is based on the median value of the 
pixels neighboring the central pixel p(c) . A 

10 conventional median filter applied to a 3 x 3 array 

would output a value corresponding to the median value 
of the nine pixels in the array. In the present case, 
it is preferred to neglect the value of the central 
pixel, since this has already been presumed to be 

15 erroneous when the second stage filtering is applied. 
Accordingly, a median value is calculated based on the 
values of the eight neighboring pixels, excluding the 
central pixel p(c) as shown in Fig. 3(b) . Since there 
is an even number of neighboring pixels, the median 

2 0 value used is the mean value of the two middle ranking 
pixel values. The sorting of the neighboring pixel 
values into rank order, described above, facilitates 
this. As seen in Fig. 5, the values of the two middle 
ranking values output from block 4 8 are summed and 

25 divided by two to provide a pseudo-median value. This 
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filtering scheme will be referred to hereinafter as a 
ring median filter and its output as the median value. 

In the example of Fig. 4, it can be seen that 
scythe (first stage) filtering and ring median (second 
5 stage filtering) both take place in parallel on the 

entire data stream. Both the scythe and median values 
are input to a final two to one multiplexer 52. The 
final output p(out) is determined by the contents of 
the defect memory 20 of Fig. 2. If the pixel location 

10 corresponding to the central pixel p(c) is stored in 
the defect memory 20, then multiplexer 52 will select 
the ring median value as the final output value. 
Otherwise, the final output value will be the scythe 
value. Since the pixel locations stored in the defect 

15 memory 20 comprise only a small proportion of the total 
number of pixels in the sensor array, scythe filtering 
will be applied to the majority of the data stream with 
ring median filtering being applied to the remainder. 

In Fig. 4, the defect memory 20 of Fig. 2 is 

2 0 represented by memory block 54 and memory management 
block 56. The pixel locations stored in the defect 
memory 2 0 are those which exhibit the most extreme 
differences from their neighbors. In the embodiment of 
Fig. 4, pixel locations are selected for inclusion in 

25 the defect memory on the basis of the magnitude of the 
difference between the value of p(c) and the scythe 
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value output from block 50. The difference between the 
two values is determined at 58 and the absolute 
magnitude of this difference at 60. The decision as to 
whether a particular pixel location should be stored 
5 can be based on a wide variety of criteria. This 

criteria is dependent in part on the size of the defect 
memory and on the memory management strategy employed. 

In the present example, a simple scheme is 
employed whereby the single worst defect in each video 

10 frame is stored in the defect memory. This defect is 
the greatest difference between the value of p(c) and 
the scythe value. For each frame, the worst defect to 
date is stored in buffer memory 62 . At the end of the 
frame, the value stored at 62 is passed to the memory 

15 block 54, together with its corresponding location in 
the sensor array. The data stored in the memory block 
54 is essentially a sorted list of pixel locations and 
associated defect magnitudes. Additional information 
could be stored if necessary. 

20 It will be understood that the beginnings and 

endings of video frames and the locations of pixels 
corresponding to pixel values in the data stream can be 
derived by the use of clocks, counters and information 
included in the data stream. This may be done in a 

25 manner which will be familiar to those skilled in the 
art. Systems for performing these functions will not 
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be described herein and are excluded from the drawings 
for the sake of clarity. 

The memory management unit 56 controls the 
output multiplexer 52 to select the ring median value 
5 as the final output when the current pixel corresponds 
to a location stored in the memory block 54. Otherwise, 
the scythe value is selected. As noted above, a 
strategy is required for managing the contents of the 
memory block 54. This is accomplished in the present 

10 example by means of a first-order auto-regression 

function also known as leaky integration. That is, the 
magnitudes of the defects stored in the memory are 
continually updated by means of the auto-regression 
formula. Once the memory 54 is full, the locations 

15 with lowest defect magnitudes can be replaced by newly 
detected defects of greater magnitude. The magnitudes 
of persistent defects will be refreshed by normal 
operation of the filtering system, while the stored 
magnitudes of transient defects will gradually 

20 attenuate until they are replaced. 

In this example, the magnitudes of stored 
defects are updated by determining the difference 
between the current pixel value p(c) and the ring 
median value at 64, and the absolute magnitude of this 

25 difference at 66. The updated value is calculated 
using the auto-regression formula at 68 from the 
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current stored value for the relevant pixel location 
and magnitude of the difference between p(c) and the 
ring median value. The stored value is updated 
accordingly. The location of the current, lowest 
5 stored value is stored in memory buffer 70 so that this 
value (MIN) can be replaced by a new defect location 
and value (MAX 62) once the memory 54 is full. 

Fig. 2 represents a generalized version of 
the preferred embodiment, employing a stored list of 

10 defect locations to apply two stage filtering to an 

incoming data stream. The first stage filtering also 
serves to determine which locations are stored. The 
second stage filtering is switched on and off on the 
basis of the stored list. As seen in Fig. 4, this 

15 functionality is implemented by applying both filtering 
functions in parallel and selecting which filter output 
to use on the basis of the stored list. The first 
stage filter output is also being employed in the 
selection of locations for storage, and the second 

20 stage filter output is being employed in the management 
of the stored list. 

Other variations of the described embodiments 
can be envisioned using different filtering functions, 
different data sampling schemes and different memory 

25 management strategies. Such variations and other 



modifications and improvements may be incorporated 
without departing from the scope of the invention. 



